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ABSTRACT 

Proposed Hi structure observatories for studying the epoch of reionization [z « 6-15) and dark 
energy (z w 6) envision compact arrays with tens of thousands of antenna elements. Fully correlating 
this many elements is computationally expensive using XF or FX correlators, and has led some groups 
to reconsider direct imaging/FFT correlators. In this paper we develop a variation of the direct 
imaging correlator we call the MOFF correlator. The MOFF correlator shares the computational 
advantages of a direct imaging correlator, while avoiding a number of its shortcomings. In particular 
the MOFF correlator makes no constraints on the antenna arrangement or type, provides a fully 
calibrated output image including widefield polarimetry and non-coplanar baseline effects, and can be 
orders-of-magnitude more efficient than XF or FX correlators for compact radio cosmology arrays. 

Subject headings: 



1. INTRODUCTION 

A new generation of radio observatories are being de- 
veloped to measure the history and evolution of our uni- 
verse using deep radio surveys of large scale structure. 
The MWA, LOFAR, and PAPER are being built to ob- 
serve the Epoch of Reionization (EoR) through large 
scale HI emission at redshifts of 6-11, and much larger 
second generation EoR observatories are being planned. 
Additionally, at redshifts of 6 HI structure experiments 
can trace large scale structure over much of the observ- 
able universe (Wyithe & Loeb 2007; Mao et al. 2008; 
Chang et al. 2008; Loeb & Wyithe 2008), enabhng a 
number of precision cosmography and dark energy mea- 
surements. In particular the dark energy baryon acous- 
tic oscillation (BAO) signal has inspired the development 
of a number of new instrument concepts, including the 
FFT Telescope (Mao et al. 2008), the Canadian Hydro- 
gen Intensity Mapping Experiment, the Cylinder Radio 
Telescope, the high frequency MWA5000 (Wyithe et al. 
2008), the Lunar Array for Radio Cosmology, and the 
Dark Ages Radio Interferometer. 

The HI cosmology and second generation Epoch of 
Reionization instruments are similar in their instrumen- 
tal characteristics, featuring compact arrangements of 
many thousands of antennas. The many short base- 
lines and wide field of view maximizes the sensitivity of 
power spectrum measurements over all scales (Morales 
2005, 2008), however, it also puts enormous processing 
demands on the correlator system. A radio correlator 
measures the cross-power correlation between all antenna 
pairs in many narrow frequency channels, and for a mod- 
ern FX correlator the computation scales as the square of 
the number of antennas. The correlator under construc- 
tion for the MWA requires 15.5 trillion complex multi- 
plies and adds per second (Tcniac;s) for 512 antennas and 
31 MHz of bandwidth. For many of the proposed radio 
cosmology instruments this quickly scales into the peta- 
fiop regime, making the correlator a dominant cost for 
these arrays. This has driven some concepts such as the 
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FFT telescope to use direct imaging correlators, despite 
their significant shortcomings, and constrain the antenna 
layout to a regular array and filled aperture. 

In this paper we introduce the MOFF (Modular Opti- 
mal Frequency Fourier) correlator concept. The MOFF 
correlator incorporates the gridding and calibration usu- 
ally associated with post-correlation image processing 
into the correlation process. This work expands on direct 
imaging correlator concepts by Daishido et al. (2000) and 
Rogers (2000) , with particular emphases on data calibra- 
tion and modular design. For the compact antenna lay- 
outs of proposed radio cosmology telescopes, the MOFF 
correlator is very efiicient, makes no constraints on the 
antenna placement, and produces a provably optimal 
data product. 

2. CORRELATOR APPROACHES 

The two most common forms of correlators are the 
XF and FX designs. The XF or lag correlator cross- 
correlates the time-domain electric field samples from 
each antenna pair using a set of time delays ('lags'), and 
then Fourier transforms the resulting time sequence of 
correlation coefficients to obtain spectral information and 
integrates to form a visibility 
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The FX or spectral-domain correlator reverses the order 
of these operations, and Fourier transforms the electric 
field samples of each antenna to create spectral time- 
sequences, which are then multiplied and integrated to 
form the visibilities 
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(The frequency transforms in Equations 1 and 2 are often 
implemented as polyphase filters to increase the spectral 
dynamic range.) Both XF and FX correlators are in wide 
production, with the VLA using an XF correlator and the 
VLBA and MWA using an FX correlator. As the number 
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of antennas increases the FX correlator begins to have 
a significant computational advantage, since the Fourier 
transform must only be performed for each antenna, not 
each antenna pair. (Please see Romney (1999) for a much 
more in-depth introduction to XF and FX correlators, 
including many subtleties which have been glossed over 
here.) 

If the antennas of an array are located on an evenly 
spaced grid,^ a third type of correlator may be used 
(Daishido et al. 2000). The direct imaging or FFT cor- 
relator (there a several names in the literature) works 
by performing a spatial Fourier transform on the electric 
field from all the antennas, creating an electric field im- 
age of the sky which can then be broken into spectra and 
squared to create an spectral-intensity map of the sky 
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(The spectral transform may be performed either before 
or after the spatial transform.) The advantage of the di- 
rect imaging correlator is that it can use a spatial FFT 
( log2 N) instead of computing each pairwise correla- 
tion {N'^), providing a large computational advantage 
for arrays contemplating tens of thousands of antennas. 
To date the largest implementation is an 8x8 array at 
Waseda University in Japan (Daishido et al. 2000). 

Direct imaging correlators have been used rarely in in- 
terferometry because they suffer a number of significant 
problems: 

1. The antennas must be arranged on a regular grid. 
This greatly limits the PSF (array beam) of the 
array, reducing the achievable dynamic range. 

2. The images suffer aliasing and other artifacts. This 

has pushed a number of the implementations to 
using very small antennas — usually single dipoles 
spaced closer than A/2 — ^to alleviate the alliasing 
effects. 

3. Calibration is very difficult, particularly for po- 
larimetry. Calibration becomes more of an issue 

as the antennas become small and the field of view 
increases. Particularly for crossed-dipoles, projec- 
tion effects lead to strong differences in sensitiv- 
ity of the two polarizations fax from zenith. This 
tendency is in direct competition with the aliasing 
problems mentioned in item 2. 

4. Individual visibilities are never formed, making de- 

convohition more difficult. 

The problem of not forming individual visibilities men- 
tioned in item 4 deserves additional attention as it plays 

a key role in the development of the MOFF correlator. 
To image the output of an FX or XF correlator, the visi- 
bilities are calibrated and gridded onto a regularly spaced 

^ Extensions to the direct imaging correlator to irregularly 
spaced arrays have been suggested (e.g. the memo by Rogers 
(2000)), but to our knowledge the implementation details of such 
a system, such as data calibration, parallelization, and the fidelity 
of the final image have not been explored in the literature. The 
MOFF correlator can be considered a specific implementation of a 
direct imaging correlator for irregular antenna spacing. 



uw-plane, and a spatial Fourier transform is used to cre- 
ate a 'dirty' image of the sky. By using an anti-aliasing 
filter (such as a spheroidal function) during the gridding 
step, the dirty image produced from FX or XF visibilities 
is superior to the image from a direct imaging correla- 
tor. Additionally, the dirty image is usually used as an 
intermediate step inside a non-linear deconvolution algo- 
rithm such as CLEAN or MEMs. In these algorithms the 
visibilities associated with a sky model are determined 
and compared to the measured values, and the model 
is improved until the visibilities converge. This kind of 
visibility-based deconvolution is not possible using the 
output of a direct imaging correlator. 

The MOFF correlator combines the Mw-based gridding 
of standard visibiUty imaging with the spatial correlation 
of a direct imaging correlator. The resulting sky image 
from a MOFF correlator is identical to the dirty image 
produced using an FX or XF correlator and is free of the 
artifacts of a direct imaging approach. In addition, the 
MOFF correlator can work with any antenna distribu- 
tion, and is computationally more efficient for compact 
arrays. 

However, for precision cosmology we have a higher bar 
than simply reproducing the dirty image of a traditional 
correlation system — we must be able to use the image 
to extract a precision model of the sky. In other words, 
we must show that we can produce an image which con- 
tains all of the sky information that is in the visibil- 
ity measurements of an FX or XF correlator. Recent 
work by Morales & Matejek (2008) and Bhatnagar et al. 
(2008) has developed the software holography/optimal 
map making approach to imaging interferometric data, 
based on CMB analysis (Tegmark 1997a, b). This ap- 
proach to creating dirty images preserves all the sky in- 
formation of the visibilities, and enables high dynamic 
range and precision polarimetric imaging. 

It should be noted that from a hardware perspec- 
tive the MOFF correlator is independent of software 
holography — it can create the image produced by any 
of the standard visibility imaging techniques^ — but soft- 
ware holography allows us to prove that the image is 
equivalent to the visibilities produced by a standard cor- 
relator. In the next section (§3) we take a brief aside to 
discuss software holography/optimal map making. The 
MOFF correlator algorithm is then presented in Section 
4, with a comparison of the computational costs in §5. 
We further develop the MOFF concept in §6 to show how 
the correlator can be broken into modular pieces (saving 
cabling and deployment costs), and then discuss the fea- 
tures and challenges of the MOFF correlator concept in 
§7. 

3. SOFTWARE HOLOGRAPHY/OPTIMAL MAP MAKING 

Recent papers by Morales & Matejek (2008) and Bhat- 
nagar et al. (2008) have introduced the software holog- 
raphy approach for analyzing interferometric radio data. 
This work extends the optimal map making approach 
widely used for the analysis of CMB data (Tegmark 
1997a,b) to interferometric data sets. Bhatnagar et al. 
(2008) have shown software holography achieves an 

This includes faceting and other advanced imaging algorithms, 
though we do not detail them here as they are largely superseded 
by the techniques of software holography. 



Fig. 1. — This figure shows the contours from a dirty map for a single baschne using software holography. For this example we have 
used an antenna consisting of a 4x4 array of dipoles with a spacing of nearly one wavelength so there are strong grating lobes, and a very 
widefield image to show both the primary beam (center) and grating sidelobes (surrounding). This is similar to the beam patterns seen 
by LOFAR and MWA antennas near the top of their frequency bands. In the image the fringe from the single visibility is clearly seen as 
the diagonal corrugations, but its amplitude has been enveloped by the known antenna pattern and the sidelobes are clearly evident. In 
traditional interferometric analysis, the corrugations would have the same amplitude across the image, as the prior of the antenna pattern 
is not used. In software holography the enveloping power pattern can vary from baseline-to-baseline to accurately represent the directional 
sensitivity of individual antenna pairs. Not shown here is the direction-dependent shifting of the fringe peaks which can be produced by 
directional differences in the phase delays of the antennas. While the beam pattern covers the sky, the corresponding uv kernel is very 
compact. 



order-of-magnitude increase in dynamic range for VLA 
imaging of crowded fields, and even more impressively 
thermal noise limited residuals of widefield polarimetric 
measurements (Bhatnagar, private coniniunication) . 

While a full review of software holography is beyond 
the scope of this paper, software holography can be 
conceptualized as changing from gridding visibilities to 
the Mu-plane with spatial delta-functions (convolved with 
anti-aliasing filters) to gridding with the holographic an- 
tenna power patterns. 

Mathematically the instrumental measurement of the 
visibilities — what the interferometer sees — can be writ- 
ten as 



m(v) = S(v, u)F(u, 6')B(6l, 6»)7(6I) + n(v). 



(4a) 



or equivalently in integral notation 



Vi = j 5{u-ni) j e-''^'''^-^B{e)I{e)(fe d^vL + Ui. 

(4b) 

Reading through the steps of Equation 4a, we start with 
the sky brightness I{9) and multiply by the antenna 
power response B. We can then Fourier transform direc- 
tional coordinates {9) to Mv-coordinates (u) which are 
then sampled at the locations of the baselines (S) and 
added to the receiver noise (n) to give a vector of the 
measured visibilities m(v). We can re-express Equa- 
tions 4a-4b by Fourier transforming the antenna power 
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response to Mf-coordinates to obtain 

m(v) = B(v, u)F(u, e)I{e) + n(v) or equiv. 



Vi = J 6{vl' 



j B(u' - u) X 



(5a) 
(5b) 
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where in the Uneax algebra version we have combined the 
antenna power response with the delta-function sampling 
function (S). Conceptually the antenna response multi- 
plication in image space (9) has been replaced with an 
equivalent convolution in ww-coordinatcs. The uv-space 
antenna power response -B(u) is given by convolving the 
direction-dependent antenna gains W 



S(u') 



u)W{u)d'^u, 



(6a) 



where the antenna gain response in uu-coordinatcs is just 
the pattern obtained from holographic antenna mapping, 
and gives the approach its name. 

Using the mathematics of CMB optimal map mak- 
ing (Tegmark 1997a,b) a software holography image is 
formed using the following procedure (Morales & Mate- 
jek 2008): 

/'(6i) =F^(6l,u)B^(u,v)N-i(v,v)m(v). (7) 



In the first step of this analysis the visibilities m(v) 
are weighted by the inverse of the system noise so that 
high signal-to-noise channels receive more weight (with 
the additional twist of allowing cross-talk to be incor- 
porated). The second step then grids the visibilities to 
a regularly sampled ut;-plane using the holographic an- 
tenna power response as the gridding kernel (and anti- 
aliasing filters as needed), followed by a Fourier trans- 
form in step three to create an image. The only differ- 
ence between this software holography analysis and tra- 
ditional imaging is the use of the antenna gain pattern 
as the gridding kernel in step 2. 

The effect of software holography can be seen by imag- 
ing the response of a single visibility as shown in Figure 
1. Effectively the fringe from each baseline has been en- 
veloped by the power response of the two antennas used 
to measure the visibility. The key gridding step is com- 
putationally efficient because the antenna response func- 
tion (W) in uu-coordinates is limited to the size of the 
antenna — naturally truncating the size of the gridding 
kernel. Other effects such as w-projection, widefield dis- 
tortions, and refractive and scintillating atmospheres can 
be easily incorporated into the software holography anal- 
ysis in Equation 7, and the interested reader is referred 
to Morales & Matejek (2008). 

A number of nice features can be proved about the 
dirty image produced by software holography, the most 
important of which is that the image is lossless — all of 
the sky information containcxl in the visibilities has been 
preserved in the software holography image (Tegmark 
1997a). Deconvolution can be performed directly on the 
software holography image — as all of the information in 
the visibilities has been preserved — and the result will be 
equivalent in quality to a visibility based deconvolution. 
For an imaging correlator targeting radio cosmology ob- 
servations, this is the result we needed: if we can produce 



the software holography image described in equation 7, 
it is equivalent in quality to the visibilities from a tradi- 
tional FX or XF correlator. 

4. THE MOFF CORRELATOR ALGORITHM 

The visibilities produced by an FX or XF correlator 

(ignoring finite time- window effects) are simply the cross 
correlations of the measured electric field: 

m{v) = {E'{a)*E\a%, (8) 

where E' are the digitized RF streams from the antennas, 
and the convolution operator (*) creates the cross-power 
products which are time averaged to produce the mea- 
sured visibilities. 

The digitized electric field seen by the correlator in 
Equation 8 can be described by 

^'(a)=W(a,r)F(r,6»)i;(6»)-Fn(a), or (9a) 
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where E is the true incident electric field and E' is the 

digitized RF stream produced by an antenna."* Equation 
9a is directly analogous to Equation 5a, where again we 
have taken the Fourier transform of the position depen- 
dent gain Wa{0) and expressed it as a convolution in 
location r (the operator W(a, r) incorporates both the 
convolution and 5-function sampling operations). This 
can be checked by showing that the correlation of elec- 
tric fields in Equation 9a reproduces Equation 5a. 

The goal of the MOFF correlator is to create an im- 
age that is equivalent to the software holography image 
in Equation 7, including all of the gridding and imaging 
steps. Starting with the software holography descrip- 
tion and the above descriptions of the measured electric 
field and correlation process we can obtain the basic the 
MOFF imaging correlation algorithm: 

I'{e) = F^(6l, u)B'^(u, v)N-i(v, v)m(v) (10a) 

= F^(e,u)B^(u,v)N-i(v,v)^^'(a) * E'*{a')) 

(10b) 

= F^(^,u)/w^(r,a)n^(a)£'(a) * (10c) 



(lOd) 



(r',a>^ (a')^'*(a'; 
F^(6',r)W^(r,a)n^(a)£;'(a) * 
F^(^',r')W^(r',a')n^(a')S'*(a') 



F^(6», r) W^(r, a) n^(a) E'{a) 

3 2 1 



(lOe) 



The first line lOa is just a repeat of the software holog- 
raphy imaging process from Equation 7, where the vis- 
ibilities are weighted by the thermal noise, gridded to 

the Mu-plane, and Fourier transformed to produce an im- 
age. In the second step (10b) we then substitute in the 

^ Quantization noise from the A/D converter is included in the 
'thermal' noise n. 
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definition of the measured visibilities in terms of the dig- 
itized antenna streams from Equation 8. Equation lOd 
in the next line then moves the thermal noise weighting 
and gridding B-^(u, v)N~^ (v, v) inside the time average, 
breaking the operations into their component parts (ie. 
Equation 6a). In this line after weighting by the ther- 
mal noise (n^) we use the gain pattern of the antenna 
W"^(r,a) to grid the electric field measurements onto a 
regularly spaced positional grid r, with the convolution 
now operating over the positional grid to produce gridded 
uw-plane measurements. In step lOe we pull the Fourier 
transform into the time average as well, and use the di- 
rectional independence of the electric field to simplify 
into our final algorithm in line lOe. 

Conceptually, the digitized electric field from each an- 
tenna E' is weighted by the thermal noise (step 1), grid- 
ded to a regularly spaced positional grid r using the 
holographic antenna gain pattern (step 2, including anti- 
aliasing filters as needed), and spatially Fourier trans- 
formed to produce an electric field image (step 3). The 
electric field image is then squared and averaged to pro- 
duce the snapshot image. A graphical representation of 
the correlation process is shown in Figure 2. 

This algorithm is the basis of the MOFF correlator. 
Conceptually what it has done is to take the visibility cal- 
ibration and gridding steps that are normally performed 
as part of the post-correlation imaging pipeline, and has 
placed them before the input to the X part of the cor- 
relator. The resulting gridded electric field can be cor- 
related using an efficient spatial FFT, greatly reducing 
the required computation for compact radio arrays. By 
incorporating the gridding as part of the correlation pro- 
cess, a MOFF correlator has a number of advantages over 
traditional imaging correlators: 

1. The antennas do not need to be placed on a regu- 
lar grid. The gridding operation produces the reg- 
ularly spaced input needed by a spatial FFT — ^the 
antennas can be arranged in any pattern. 

2. The grid spacing can be chosen to match the FOV 
and science case, using the same arguments used to 
determine the ut;-grid spacing in standard imaging. 

3. The antenna signal is calibrated using the holo- 
graphic gain pattern so antennas of any size and 
type can be used — ^we are no longer constrained to 
using A/2 dipoles. Correspondingly the wu-spacing 
of the grid can be much larger, greatly reducing the 
correlation load for many arrays. 

4. The output image has the equivalent information 
of the FX or XF correlator visibilities, allowing pre- 
cision deconvolution and polarimetry. In addition, 
some of the artifacts seen in traditional imaging 
correlators such as non-coplanar baselines and anti- 
aliasing can be mitigated by using the appropriate 
gridding kernel within the MOFF correlator. 

There are a number of implementation subtleties that 
have been worked through, but are not detailed here. For 
example, for steerable antennas phase delays should be 
added to the signal just prior to gridding to orient the 
gridded input to the look direction. However it is more 
computationally convenient to tie the east-west orienta- 
tion of this input grid to the ground and do synthesis 



rotation in the image plane than to rotate the grid with 
the sky as is more traditionally done. Other imaging ef- 
fects can be readily incorporated, including w-projection 
for non-coplanar baselines, widefield distortions, image 
oversampling, and widefield refractive and scintillating 
atmospheric distortions. It is also worth noting again 
that any kind of gridding can be used in the MOFF 
correlator — it does not depend on software holography. 
Software holography ensures that the image contains all 
the information in the visibilities, but any standard imag- 
ing procedure can be used. 

5. COMPUTATIONAL COMPARISON 

In this section we compare the computational require- 
ments of the MOFF correlator to FX and imaging cor- 
relators. It has been shown that FX correlators are 
nearly always more efficient than XF correlators (Rom- 
ney 1999), so we will not compare them here (XF corre- 
lators can be more flexible, which is why they can make 
more sense for arrays like the VLA). In addition we will 
follow the tradition of neglecting the computation in the 
frequency (F) part of the correlation, as it small com- 
pared to the spatial correlation requirements. 

Both the FX and MOFF correlators start with a 
Fourier transform in the frequency domain, with the spa- 
tial correlation then proceeding within the narrow fre- 
quenc:y c:liannels. For an FX correlator the number of 
complex multiplications and additions in the spatial part 
of the calculation (X) is equal to the bandwidth times the 
number of antennas Na squared: 



FX^BfNl 



(11) 



(We've subscripted bandwidth with / to avoid confusion 
with the antenna power response pattern referred to as B 

earlier.) For the MOFF correlator the two dimensional 
spatial FFT scales with the number of grid locations Ng-. 



MOFF oc BfCt Ng logs 



(12) 



(The square root comes from assuming a square grid with 
the length of one side equal to and the constant Ct 

ranges between 1/2 and 1 depending on the implemen- 
tation of twiddle FFT algorithms (Brigham 1974).) The 
interesting feature of the MOFF correlation load is that it 
is independent of the number of antennas — adding more 
antennas does not change the number of grid points. 

The parameter space of array size and antenna density 
over which the MOFF correlator is more efficient than an 
FX correlator can be described following an argument by 
Aaron Parsons (personal communication). If the array 
filling factor is defined so the number of antennas is 



J^Ng, the MOFF correlator will be more efficient if 
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(13) 



Ignoring the logarithm as a relatively small term, we can 
rearrange to obtain 



^ > 
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(14) 



As the grid spacing is often chosen to cover the primary 
response of the antennas, the number of grid points is 
typically the square of the ratio of the array diameter 
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{Da) to the antenna diameter {da}- Substituting we ob- 
tain 



Equations 14 and 15 both show that as the array be- 
comes larger in size, a smaller filling factor is needed for 
the MOFF correlator to be a viable alternative. An FX 
correlator will be more efficient for sparse arrays, because 
it does not calculate any correlations where there are no 
antennas. However, as the array filling factor becomes 
large the computational efficiency of the FFT starts to 
give the MOFF correlator a significant advantage. This 
means that the FX correlator is much more efficient for 
sparse arrays such as the VLA, but is less efficient for 
future cosmology focused arrays such as the MWA5000, 
LARC, FFT Telescope, and possibly the SKA. 

For the MWA with 500 antennas in a 1.5 km array, 
the FX correlator under construction will perform 15.5 
Tcmacs (trillions of complex multiply and additions per 
second, 31 MHz x 125000 baselines x 4 polarizations). 
An equivalent MOFF correlator would require 4-5 times 
as many calculations. However, for the proposed MWA 
upgrade to MWA5000 with ten times the number of an- 
tennas in the same area, the MOFF correlator would re- 
quire approximately 20 times less computation. For the 
Lunar Array for Radio Cosmology (LARC), the MOFF 
correlator would be more than two orders-of-magnitude 
more efficient than an FX design. 

The comparison to a direct imaging correlator is trick- 
ier, because the direct imaging correlator requires a filled 
aperture with the antennas on a regularly spaced rectan- 
gular grid. For a filled regularly spaced array the compu- 
tational efficiency is nearly identical between the direct 
imaging correlator and the MOFF correlator, though the 
image produced by the MOFF correlator is markedly su- 
perior. With the MOFF correlator the spacing of the 
grid is independent from the antenna spacing, and can 
be adjusted to optimize the FOV of the resulting im- 
age. In many cases this can reduce the number of input 
nodes, resulting in a more efficient calculation. However, 
the real advantage of the MOFF correlator is the ability 
to place the antennas in any arrangement and the ability 
to accurately calibrate data from large antenna elements 
that span many wavelengths. 

6. MODULAR DESIGN 

The last step in describing the MOFF correlator is par- 
allelizing the computation. One could use the same ap- 
proach as an FX correlator and parallelize by frequency. 
In this approach one X board handles all of the antennas 
(or spatial nodes) for a small range of frequencies. The 
complication is the 'corner turn' between the F and X 
stages: the output of the F stage boards is all the fre- 
quencies for one antenna, and the input of the X stage 
boards is all the antennas over a small subset of the fre- 
quencies. Much of the design of modern FX correlators 
is concerned with efficiently performing this corner turn 
to allow parallelization (Urry et al. 2007: Parsons et al. 
2008). 

Due to the structure of the spatial FFT, there is an 
alternate approach to parallelizing the computation of 
the MOFF correlator which may offer some distinct ad- 
vantages. The spatial FFT can be separated into spatial 
units we are calling 'modules,' as shown in Figure 2. Each 



module would calculate a portion of the full FFT for all 
of the frequencies. For the first few stages of the FFT 
this approach will require horizontal communications be- 
tween the modules to pass intermediate results through 
the array (row d). The later stages of the FFT are all con- 
tained within one module (row e) and can be performed 
without additional modulc-to-module communication. 

In the module approach to parallelization, inter- 
module communication is required at several different 
stages. In the gridding stage (row b), the kernel used to 
grid can overlap the boundary of adjacent modules. In 
this case a copy of the digital antenna signal needs to 
be provided to the neighboring module (s). The gridding 
kernels are usually about the size of the antennas in ex- 
tent, so the fraction of antennas which straddle module 
boundaries tends to be small. Inter-module communi- 
cation is also required in the first stages of the spatial 
FFT where intermediate solutions must be exchanged, 
as shown in Figure 2. The ratio of communication to 
computation can be optimized by adjusting the portion 
of the array handled by one module. The computational 
requirements of a module grow in relation to the area 
of the array handled (number of input nodes), while the 
communication requirements grow as the circumference 
of that area.^ 

Breaking the correlator into spatial modules could sig- 
nificantly reduce the cost of radio cosmology arrays. In 
costing out radio cosmology arrays, the cost of running 
cable from the antennas to the correlator is surprisingly 
large and can often drive the array design. In the MWA 
the antenna signals are digitized and collected in the field 
so they can be multiplexed onto a small number of digital 
fibers, and similar approaches are used for LOFAR and 
LWA. For the MOFF correlator we could include the cor- 
relator modules in the field-based digital receivers, so the 
correlator does not need to be located in a central build- 
ing. This could reduce the cabling to only neighbor-to- 
neighbor connections between receiver/correlator mod- 
ules. 

In addition to cost savings, this could help the phased 
build-out of an array. As each section of the array is 

completed and connected to its neighbors, it immediately 
becomes correlated and enhances the scientific reach of 
the observatory. Thus an array could be built in stages 
over a number of years, gradually increasing in capability. 
This growth of capability is much more difficult for an 
FX correlator due to the corner turn and the tremendous 
growth of cross-products (antennas squared). 

The modularity of the MOFF correlator does not c;oine 
without a cost, as there is significant latency in inter- 
module communications. These delays can be hidden by 
pipelining the computation so other correlation work is 
performed while the intermediate FFT products are en 
route. This pipelining and communication architecture 
is a major area of continuing development. 

7. DISCUSSION 

The MOFF correlator concept has a number of advan- 
tages over traditional correlator designs for envisioned 

^ There are also mixed-mode parallelization options, where the 
correlator is parallelized by frequency within one module (multi- 
ple boards) but broken down spatially across the array, with the 
advantage of a much smaller corner turn. 
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radio cosmology observatories. The advantages we have 
detailed include: 

• Calibrated output image, equivalent in quality to 

FX visibilities (§3). 

• More efficient than FX or XF correlators for com- 
pact arrays with very large numbers of antennas, 
sometimes by orders-of-magnitude (§5). 

• No constraints on antenna placement or type (§4) . 

• Modular installation and reduced cabling costs 

(§6). 

A couple of features we have not discussed but which 
follow directly from the MOFF design are: 

• Electric field image can be tapped for pulsar studies 
and transient detection. Each pixel is effectively a 

calibrated tied-array beam. 

• Hybrid modes arc possible, where a MOFF correla- 
tor is used for all of the antennas in a compact sta- 
tion and the electric field image from selected pixels 
is sent to an FX correlator for high angular reso- 
lution interferometry between the stations. Simi- 
lar hybrid modes may be useful for phased-array 
feeds, where a MOFF correlator would turn the 
image back into calibrated spatial Fourier modes 
for widefield FX correlation [citation of eigenvalue 
correlation]. 
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As with any design there are some disadvantages as 
well. The primary disadvantage of the MOFF correlator 
is that the calibration must be applied as part of the cor- 
relation process — one cannot re-calibrate the data dur- 
ing post-processing. For very large cosmology arrays, the 
overwhelming number of antenna pairs may make saving 
raw visibility data impractical anyway. The MWA does 
not save the visibility information for this reason and 
instead performs realtime calibration and imaging, and 
future cosmology arrays will probably do the same. How- 
ever, this may limit the usefulness of the MOFF concept 
in certain applications. 

A second disadvantage is that the holographic antenna 
gain patterns needed for calibration (§3) cannot be ob- 
tained directly from the image produced by the MOFF 
correlator. Effectively the antenna beam patterns have 
been co-added during the correlation process (most eas- 
ily seen in Equation 10a), and one needs the individual 
visibilities to measure the holographic antenna gain re- 
sponse. As part of real world implementations of the 
MOFF correlator, digitized signals from selected anten- 
nas will need to be pulled out of the MOFF correlator 
(Figure 2 row b) so holographic gain patterns can be 
formed offline. 
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